Skip to content

feat(ci): swap Python's OTLP encode+upload for the native Rust pipeline#1466

Open
jd wants to merge 1 commit into
mainfrom
devs/jd/worktree-rust-port/encode-junit-cases-otlp-upload-mergify-ci-insights--cbc72716
Open

feat(ci): swap Python's OTLP encode+upload for the native Rust pipeline#1466
jd wants to merge 1 commit into
mainfrom
devs/jd/worktree-rust-port/encode-junit-cases-otlp-upload-mergify-ci-insights--cbc72716

Conversation

@jd
Copy link
Copy Markdown
Member

@jd jd commented May 27, 2026

Second layer of the mergify ci junit-process port — replace the
opentelemetry-exporter-otlp-proto-http encode-and-upload path
in mergify_cli/ci/junit_processing/upload.py with a call into
a new hidden Rust subcommand, mergify _internal junit-upload,
that re-parses the JUnit XML files, builds the OTLP
ExportTraceServiceRequest with the quarantine set baked in,
gzips it, and POSTs it to
/v1/repos/<owner>/<repo>/ci/traces.

Two new modules under junit_process:

  • spans builds one session span per upload, one suite span per
    <testsuite>, and one case span per <testcase>, all sharing
    a resource that carries the CI-env attributes the backend uses
    for routing (provider, pipeline run id, branch, head SHA, …).
    The test cases pin parent/child wiring, status-code mapping,
    attribute propagation, and resource scraping by feeding a
    deterministic RNG into a hidden seam. UploadMetadata carries
    an optional run_id (the bridge passes the same 16-char hex
    Python already printed to its UI; decoded to 8 bytes as the
    session span id) and a quarantined name set (each matching
    case span gets cicd.test.quarantined = true).

  • upload POSTs the request with the same headers the Python
    OTLPSpanExporter used to send (Bearer auth,
    application/x-protobuf, Content-Encoding: gzip) and
    matches the error wording so any log scrapers tracking Python
    output keep working. Wiremock covers the happy path, the
    empty-request short-circuit, and the 401 error surface.

The detector grows the resource-attribute lookups Python emits
(pipeline name, job name, run id/attempt/url, head/base ref,
head SHA, repository URL, runner name). On GitHub Actions
pull_request builds, GITHUB_SHA points at the synthetic merge
commit GitHub creates by pre-merging the PR head into the base —
not the actual code the tests ran against. The contributor's real
head SHA lives at pull_request.head.sha inside GITHUB_EVENT_PATH
and that is the value the dashboards correlate with, so
get_head_sha prefers the event payload and falls back to
GITHUB_SHA when the payload is missing, malformed, or the build
isn't a PR event. Three unit tests pin the precedence: payload
wins, missing-PR-field falls back, missing-event-file falls back.
The CircleCI PR-build API fallback Python implements stays
Python-side for now — it requires a GitHub REST API client we
don't have on the Rust side yet.

Python's upload.upload(api_url, token, repository, spans=...)
becomes
upload.upload(api_url, token, repository, files=..., run_id=..., quarantined_names=..., test_framework=..., test_language=...).
The caller (process_junit_files) extracts quarantined_names
from the spans the Python quarantine pass already mutated
(cicd.test.quarantined is True) so the Rust span builder
reproduces the same wire shape the Python pipeline used to emit.
The opentelemetry-exporter-otlp-proto-http runtime dep is
dropped — opentelemetry-sdk stays because the quarantine and
report paths still consume ReadableSpan attributes (those move
to native in Phase C).

opentelemetry-proto is the only otel dep we pull in on the
Rust side, gated to gen-tonic-messages + trace so it boils
down to the prost generated types plus the trace proto module —
no tonic, no otel SDK, no exporter runtime. Compression is
flate2 and the HTTP layer reuses the existing workspace
reqwest to keep the TLS / rustls flavour consistent.

This is a transitional bridge. When Phase C of the port lands
later in the stack (the native ci junit-process orchestrator
that subsumes the entire command), the Python side of
junit-process and the _internal junit-upload subcommand
both go away. Cleanup checklist for that follow-up:

  • delete InternalSubcommand::JunitUpload,
    InternalJunitUploadArgs, InternalJunitUploadOpts,
    NativeCommand::InternalJunitUpload, and the
    ("_internal", "junit-upload") entry in NATIVE_COMMANDS
  • delete mergify_cli/ci/junit_processing/upload.py
  • drop the remaining opentelemetry-sdk Python dep once the
    rest of process_junit_files is gone

Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com

@jd
Copy link
Copy Markdown
Member Author

jd commented May 27, 2026

This pull request is part of a Mergify stack:

# Pull Request Link
1 feat(ci): swap Python's OTLP encode+upload for the native Rust pipeline #1466 👈
2 feat(ci): promote ci junit-process and its junit-upload alias to native Rust #1467

@mergify
Copy link
Copy Markdown
Contributor

mergify Bot commented May 27, 2026

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🔴 👀 Review Requirements

Waiting for

  • #approved-reviews-by>=2
This rule is failing.
  • any of:
    • #approved-reviews-by>=2
    • author = dependabot[bot]
    • author = mergify-ci-bot
    • author = renovate[bot]

🔴 🔎 Reviews

Waiting for

  • #review-requested = 0
This rule is failing.
  • #review-requested = 0
  • #changes-requested-reviews-by = 0
  • #review-threads-unresolved = 0

🟢 🤖 Continuous Integration

Wonderful, this rule succeeded.
  • all of:
    • check-success=ci-gate

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert|ui)(?:\(.+\))?:

🟢 📕 PR description

Wonderful, this rule succeeded.
  • body ~= (?ms:.{48,})

@jd jd force-pushed the devs/jd/worktree-rust-port/encode-junit-cases-otlp-upload-mergify-ci-insights--cbc72716 branch from 3a3d6cc to ee06c7a Compare May 27, 2026 08:27
@jd jd force-pushed the devs/jd/worktree-rust-port/parse-junit-xml-reports-native-rust--7d0fb778 branch from 380699c to aaf35b0 Compare May 27, 2026 08:27
@jd jd temporarily deployed to func-tests-live May 27, 2026 08:27 — with GitHub Actions Inactive
@jd jd temporarily deployed to func-tests-live May 27, 2026 08:27 — with GitHub Actions Inactive
@jd
Copy link
Copy Markdown
Member Author

jd commented May 27, 2026

Revision history

# Type Changes Reason Date
1 initial 3a3d6cc 2026-05-27 08:27 UTC
2 content 3a3d6cc → ee06c7a (raw) 2026-05-27 08:27 UTC
3 rebase ee06c7a → 0c646f7 (rebase only) 2026-05-27 08:46 UTC
4 content 0c646f7 → fa0eeaf (raw) 2026-05-27 09:18 UTC
5 content fa0eeaf → dd1bfdf Move the GITHUB_OUTPUT scrub out of this commit into the scrubber-consolidation commit lower in the stack, so the fix covers every PR (including #1465) instead of only the ones at or above this one. 2026-05-27 10:04 UTC
6 rebase dd1bfdf → 2ecb8c2 (rebase only) 2026-05-27 10:15 UTC
7 content 138ac43 → e901cae (raw) Fold the GitHub Actions PR-event head-SHA fix back into this commit (was previously a separate follow-up #1469). The split was wrong — get_head_sha was introduced here with the incomplete implementat… 2026-05-28 08:20 UTC
8 content e901cae → 0e3e818 (raw) Update sample_parsed in spans tests to construct ParseResult with the new suite_names field, introduced in the Phase A swap to track suite document-order for the Python bridge. 2026-05-28 09:45 UTC
9 content 0e3e818 → 19c0db2 (raw) Refactored Phase B from 'add inert Rust library code for OTLP encode+upload' into 'swap Python's opentelemetry-exporter-otlp-proto-http call for the Rust upload pipeline via a hidden _internal junit-… 2026-05-28 10:21 UTC
10 content 19c0db2 → 0963560 (raw) Pass ruff check --fix over test_upload.py to add the COM812 trailing commas ruff format collapsed; the CI workflow runs check after format and treats COM812 as a hard error even though the formatter … 2026-05-29 06:35 UTC
11 content 0963560 → 63a2a41 (raw) Switch the test_upload.py mock targets from attribute access (upload.junit / upload.subprocess) to string-form patch paths so mypy's strict attr-defined check doesn't reject them — those modules are … 2026-05-29 08:31 UTC
12 rebase 63a2a41 → 6faeb52 (rebase only) 2026-05-29 11:32 UTC
13 rebase 6faeb52 → 624158f (rebase only) 2026-05-29 14:03 UTC

@mergify mergify Bot had a problem deploying to Mergify Merge Protections May 27, 2026 08:28 Failure
@jd jd force-pushed the devs/jd/worktree-rust-port/encode-junit-cases-otlp-upload-mergify-ci-insights--cbc72716 branch from ee06c7a to 0c646f7 Compare May 27, 2026 08:46
@jd jd temporarily deployed to func-tests-live May 27, 2026 08:46 — with GitHub Actions Inactive
@mergify mergify Bot had a problem deploying to Mergify Merge Protections May 27, 2026 08:46 Failure
@mergify mergify Bot requested a review from a team May 27, 2026 08:59
@jd jd force-pushed the devs/jd/worktree-rust-port/encode-junit-cases-otlp-upload-mergify-ci-insights--cbc72716 branch from 0c646f7 to fa0eeaf Compare May 27, 2026 09:18
@jd jd changed the title feat(ci): encode JUnit cases as OTLP and upload to Mergify CI Insights feat(ci): swap Python's OTLP encode+upload for the native Rust pipeline May 28, 2026
@jd jd force-pushed the devs/jd/worktree-rust-port/encode-junit-cases-otlp-upload-mergify-ci-insights--cbc72716 branch from 0e3e818 to 19c0db2 Compare May 28, 2026 10:21
@jd jd temporarily deployed to func-tests-live May 28, 2026 10:21 — with GitHub Actions Inactive
@jd jd temporarily deployed to func-tests-live May 28, 2026 10:21 — with GitHub Actions Inactive
@mergify mergify Bot had a problem deploying to Mergify Merge Protections May 28, 2026 10:21 Failure
@jd jd force-pushed the devs/jd/worktree-rust-port/parse-junit-xml-reports-native-rust--7d0fb778 branch from 63399a1 to 73fc577 Compare May 29, 2026 06:35
@jd jd force-pushed the devs/jd/worktree-rust-port/encode-junit-cases-otlp-upload-mergify-ci-insights--cbc72716 branch from 19c0db2 to 0963560 Compare May 29, 2026 06:35
@jd jd temporarily deployed to func-tests-live May 29, 2026 06:35 — with GitHub Actions Inactive
@jd jd temporarily deployed to func-tests-live May 29, 2026 06:35 — with GitHub Actions Inactive
@mergify mergify Bot had a problem deploying to Mergify Merge Protections May 29, 2026 06:35 Failure
@jd jd force-pushed the devs/jd/worktree-rust-port/encode-junit-cases-otlp-upload-mergify-ci-insights--cbc72716 branch from 0963560 to 63a2a41 Compare May 29, 2026 08:31
@jd jd force-pushed the devs/jd/worktree-rust-port/parse-junit-xml-reports-native-rust--7d0fb778 branch from 73fc577 to 64fccaf Compare May 29, 2026 08:31
@jd jd temporarily deployed to func-tests-live May 29, 2026 08:31 — with GitHub Actions Inactive
@jd jd temporarily deployed to func-tests-live May 29, 2026 08:31 — with GitHub Actions Inactive
@mergify mergify Bot had a problem deploying to Mergify Merge Protections May 29, 2026 08:31 Failure
@jd jd force-pushed the devs/jd/worktree-rust-port/parse-junit-xml-reports-native-rust--7d0fb778 branch from 64fccaf to f6a515e Compare May 29, 2026 11:32
@jd jd force-pushed the devs/jd/worktree-rust-port/encode-junit-cases-otlp-upload-mergify-ci-insights--cbc72716 branch from 63a2a41 to 6faeb52 Compare May 29, 2026 11:32
@jd jd temporarily deployed to func-tests-live May 29, 2026 11:32 — with GitHub Actions Inactive
@jd jd temporarily deployed to func-tests-live May 29, 2026 11:32 — with GitHub Actions Inactive
@jd jd temporarily deployed to func-tests-live May 29, 2026 11:32 — with GitHub Actions Inactive
@mergify mergify Bot had a problem deploying to Mergify Merge Protections May 29, 2026 11:33 Failure
Base automatically changed from devs/jd/worktree-rust-port/parse-junit-xml-reports-native-rust--7d0fb778 to main May 29, 2026 14:00
@mergify
Copy link
Copy Markdown
Contributor

mergify Bot commented May 29, 2026

@jd this pull request is now in conflict 😩

@mergify mergify Bot added the conflict label May 29, 2026
Second layer of the `mergify ci junit-process` port — replace the
`opentelemetry-exporter-otlp-proto-http` encode-and-upload path
in `mergify_cli/ci/junit_processing/upload.py` with a call into
a new hidden Rust subcommand, `mergify _internal junit-upload`,
that re-parses the JUnit XML files, builds the OTLP
`ExportTraceServiceRequest` with the quarantine set baked in,
gzips it, and POSTs it to
`/v1/repos/<owner>/<repo>/ci/traces`.

Two new modules under `junit_process`:

- `spans` builds one session span per upload, one suite span per
  `<testsuite>`, and one case span per `<testcase>`, all sharing
  a resource that carries the CI-env attributes the backend uses
  for routing (provider, pipeline run id, branch, head SHA, …).
  The test cases pin parent/child wiring, status-code mapping,
  attribute propagation, and resource scraping by feeding a
  deterministic RNG into a hidden seam. `UploadMetadata` carries
  an optional `run_id` (the bridge passes the same 16-char hex
  Python already printed to its UI; decoded to 8 bytes as the
  session span id) and a `quarantined` name set (each matching
  case span gets `cicd.test.quarantined = true`).

- `upload` POSTs the request with the same headers the Python
  `OTLPSpanExporter` used to send (`Bearer` auth,
  `application/x-protobuf`, `Content-Encoding: gzip`) and
  matches the error wording so any log scrapers tracking Python
  output keep working. Wiremock covers the happy path, the
  empty-request short-circuit, and the 401 error surface.

The detector grows the resource-attribute lookups Python emits
(pipeline name, job name, run id/attempt/url, head/base ref,
head SHA, repository URL, runner name). On GitHub Actions
`pull_request` builds, `GITHUB_SHA` points at the synthetic merge
commit GitHub creates by pre-merging the PR head into the base —
not the actual code the tests ran against. The contributor's real
head SHA lives at `pull_request.head.sha` inside `GITHUB_EVENT_PATH`
and that is the value the dashboards correlate with, so
`get_head_sha` prefers the event payload and falls back to
`GITHUB_SHA` when the payload is missing, malformed, or the build
isn't a PR event. Three unit tests pin the precedence: payload
wins, missing-PR-field falls back, missing-event-file falls back.
The CircleCI PR-build API fallback Python implements stays
Python-side for now — it requires a GitHub REST API client we
don't have on the Rust side yet.

Python's `upload.upload(api_url, token, repository, spans=...)`
becomes
`upload.upload(api_url, token, repository, files=..., run_id=...,
quarantined_names=..., test_framework=..., test_language=...)`.
The caller (`process_junit_files`) extracts `quarantined_names`
from the spans the Python quarantine pass already mutated
(`cicd.test.quarantined is True`) so the Rust span builder
reproduces the same wire shape the Python pipeline used to emit.
The `opentelemetry-exporter-otlp-proto-http` runtime dep is
dropped — `opentelemetry-sdk` stays because the quarantine and
report paths still consume `ReadableSpan` attributes (those move
to native in Phase C).

`opentelemetry-proto` is the only otel dep we pull in on the
Rust side, gated to `gen-tonic-messages + trace` so it boils
down to the prost generated types plus the trace proto module —
no tonic, no otel SDK, no exporter runtime. Compression is
`flate2` and the HTTP layer reuses the existing workspace
`reqwest` to keep the TLS / rustls flavour consistent.

This is a transitional bridge. When Phase C of the port lands
later in the stack (the native `ci junit-process` orchestrator
that subsumes the entire command), the Python side of
`junit-process` and the `_internal junit-upload` subcommand
both go away. Cleanup checklist for that follow-up:
- delete `InternalSubcommand::JunitUpload`,
  `InternalJunitUploadArgs`, `InternalJunitUploadOpts`,
  `NativeCommand::InternalJunitUpload`, and the
  `("_internal", "junit-upload")` entry in `NATIVE_COMMANDS`
- delete `mergify_cli/ci/junit_processing/upload.py`
- drop the remaining `opentelemetry-sdk` Python dep once the
  rest of `process_junit_files` is gone

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Change-Id: Icbc727166711d76678877aa172ca47c2dcc07ebc
@jd jd force-pushed the devs/jd/worktree-rust-port/encode-junit-cases-otlp-upload-mergify-ci-insights--cbc72716 branch from 6faeb52 to 624158f Compare May 29, 2026 14:03
@jd jd temporarily deployed to func-tests-live May 29, 2026 14:03 — with GitHub Actions Inactive
@jd jd temporarily deployed to func-tests-live May 29, 2026 14:03 — with GitHub Actions Inactive
@mergify mergify Bot had a problem deploying to Mergify Merge Protections May 29, 2026 14:03 Failure
@mergify mergify Bot removed the conflict label May 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant